Vector space representation of language probabilities through SVD of n-gram matrix

In this paper we introduce the vector space representation of the N-gram language model where vectors of K dimensions are given to both words and contexts, i.e., an N-1 word sequence, so that the scalar product of a ‘word vector’ and a ‘context vector’ gives the corresponding N-gram probability. The vector space representation is obtained from singular value decomposition (SVD) of the co-occurrence frequency matrix (CFM) of the context and the word. The effectiveness of the proposed representation is examined by determining how the number of N-gram parameters can be reduced through clustering and truncation of the dimensions defined on the given vector space. From the experimental results, it is confirmed that the number of model parameters can be reduced to less than 17.5% of the original number of model parameters and the proposed method is more effective than the word clustering method based on mutual information.